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Background 


The consultation on REF 2021 is open until October 15"" 2018. Two documents have been published, Draft 
Guidance and Draft Panel Criteria and Working Methods. 


We advise that you take a quick look at both these documents in advance on the workshop on Friday, to 
familiarise yourself with their content. On the next two slides we provide a quick overview of their focus and 
of the consultation questions that are posed within them. 


The focus of the workshop is on the guidance about public engagement in both documents, and on what (if 

anything) could be changed or added to improve this. By improve, we mean: 

e Make it clearer 

e Make it more coherent and consistent 

e Make it more ‘appropriate’ (a term used consistently in the consultation. We are interpreting ‘appropriate’ 
to mean in accordance with good practice, and realistic within the constraints of the REF exercise) 


In fact, there are very few explicit mentions of public engagement in the two documents, although what is 
included offers strong encouragement to universities to consider including impacts arising from public 
engagement in their submitted case studies. Slide 4 includes the key references from both documents. 


The purpose of this document is to foreground what we consider to be the most significant issues with the 
clarity, appropriateness an consistency of the guidance. Key recommendations we will be outlined about how 
these might be addressed in revisions to the guidance. 


We will present the rationale for and some of the detail of our proposed response at the workshop on the 14t" 
September. 


We invite you to come prepared to contribute to and build on our current thinking and approach. Throughout 
this briefing document we have highlighted prompt questions which we hope you will have time to consider in 
advance. We also invite you to bring references to or examples of frameworks, tools or resources which you 
think we could usefully include in our response. We also invite you to come prepared to share other issues 
and challenges with the guidance which we haven’t foregrounded, which you think it is important to address 
during the consultation. 
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2018/01 Draft guidance 2018102 Consultation on 


July 2018 July 2018 


on submissions the draft panel 
criteria and 


working methods 


Research 
Excellence 
Framework 


Previous consultations and reviews of PE in the REF 
There has been lots of activity to shape and inform the framing of 
public engagement in the REF, and you can access an overview and 
links here on the NCCPE website. 


Of particular note are: 

e The NCCPE' review of Public Engagement in the 2014 REF, which 
analysed the database of impact case studies and impact 
templates to examine how public engagement featured and to 
draw oud lessons and recommendations to inform REF 2021 
The NCCPE’s response to the 2017 consultation on the REF 
The briefing paper and write up of a workshop hosted by the REF 
team in February 2018 which explored how public engagement 
should feature in the guidance for REF 2021 


218101 Draft guidance 


July 2018 


on submissions 


Research 
Excellence 
Framework 


Open access polity... 

Part 3 Section 1: Staff details (REF 1a/b) 

Part 3 Section 3: impact (REF3) 

Part 3 Section 4: Environment data (REF4AANG) ..................o.oooooooooooooo 
Part 3 Section 5: Environment (REF Sa/b) ............ 


Executive summary 


Purpose 

1. This document is for consultation. The document: 
sets out the draft framework and generic criteria for assessment in the 2021 
Research Excellence Framework (REF) 
specifies the content, data requirements and related definitions for 
submissions to REF 2021 
guides higher education institutions (HEIs) on policy and practical matters in 
preparing submissions. 


2018/02 
July 2018 


The two consultation documents 


Consultation on 


the draft 


panel 


criteria and 
working methods 


REF 2021 


Annex B: Summary of additional information about outputs 

Annex C: Main Panel D — output types & submission guidance. 
stay Annex D: Managing conflicts of interest 
a Annex E: Confidentiality and information security agreement for REF 2021 paneis. 
Annex F: List of abbreviations 


Executive summary 
Purpose 


1. This document sets out the draft assessment criteria and working methods of the main and 
sub-panels for the Research Excellence Framework (REF) 2021, for consultation. 


2. The final panel criteria and working methods will be published in early 2019. This document, 
taken together with the ‘Guidance on submissions’, also published in draft for consultation (REF 
2018/01), will provide a comprehensive description of the information required in submissions to the 
REF, and how the REF panels will assess submissions. 


Key points 


3. The REF is a process of expert review. Expert sub-panels for each of 34 units of assessment 
will carry out the assessment, working under the leadership and guidance of four main panels. 


4. UK higher education institutions (HEIs) will be invited to make submissions by 27 November 
2020. The REF main and sub-panels will assess submissions during 2021, and results will be 
published in December 2021. The results will inform the allocation of research funding by the UK 
higher education funding bodies, from 2022-23. 


5. This document sets out a draft combined statement of criteria and working methods across the 
four main panels, for consultation. Supplementary criteria are set out for each main panel where 
applicable. The main panels’ supplementary criteria are intended to be read alongside the combined 
criteria, and do not replace it 3 


You can access a word document that lists all the questions here, although answers need to be submitted via The consultation questions 
their online platform. Many of the topics being consulted on are specific and technical. We have italicised and 
highlighted in red those questions where we intend to make a response 


Questions about the Draft Guidance 

1. Is the guidance clear in 'Part 1: Overview of the assessment framework‘? 

2. Is the guidance clear in 'Part 2: Submissions' 

3. Is the guidance is clear in 'Part 3, Section 1: Staff details‘? 

4. Possible indicators of research independence are set out at paragraph 130, including a reference to a list of independent fellowships. This list is intended to guide institutions on 
determining independence for staff holding fellowships from major research funders. The list is not intended to be comprehensive. Do you have any comments on the clarity, usefulness, or 
coverage of this list? 

5. Do you agree with the proposed eligibility of seconded staff set out at paragraphs 121.c to d? 

6. Do you agree with the proposed ineligibility of staff based in a discrete department or unit outside the UK? 

7. Will the proposed approach for taking account of circumstances achieve the aim of promoting equality and diversity in REF 2021? 

8. Is the guidance in 'Part 3, Section 2: Research outputs’ clear? 

9. A glossary of output types and collection formats is set out at Annex K, to provide increased clarity to institutions on categorising types of output for submission. Do you have any 
comments on the clarity and usefulness of this annex? 

10. Paragraph 206.b sets out the funding bodies’ intention to make ineligible the outputs of former staff who have been made redundant (except where the staff member has taken voluntary 
redundancy).Do you agree with this proposal? 

11. Do you agree with the proposed intention to permit the submission of co-authored outputs only once within the same submission? 

12. . How feasible do you consider to be the approach set out at paragraphs 267 to 271 for capturing information on the balance of research activity of different costs within submitting units 
in UOA 4? (300 word limit) 

13. Is the guidance in 'Part 3, Section 3: Impact' clear? 

14. Is the guidance in 'Part 3, Section 4: Environment data' clear? 

15. Is the guidance in 'Part 3, Section 5: Environment' clear? 

16. Please provide any further comments on the 'Guidance on submissions’, including Annexes A-M 


Questions about the Draft Criteria and working methods 

1. Do the UOA descriptors provide a clear and appropriate description of the disciplines covered by the UOAs? Please include any suggestions for refining the descriptors and state which 
UOA(s) you are commenting on 

2. Are the criteria clear are appropriate in 'Part 3, Section 1: Submissions‘? 

3. Are the criteria clear and appropriate in 'Part 3, Section 2: Outputs‘? 

4. Are the criteria clear and appropriate in 'Part 3, Section 3: Impact‘? 

5. Are the criteria clear and appropriate in 'Part 3, Section 5: Environment‘? 

6. Are the criteria clear and appropriate in 'Part 4: Panel procedures‘? 

7. Are the criteria clear and appropriate in 'Part 5: Panel working methods‘? 

8. Overall, do the ‘Panel criteria and working methods’ achieve an appropriate balance between consistency and allowing for discipline-based differences between the panels? 


Explicit references to public engagement in the draft guidance aneng Consultation on 
the draft panel 


criteria and 
working methods 


2018/0 Draft guidance 
July 2018 . . 
on submissions 


318. There are many ways in which research may have 
underpinned impact, including 
but not limited to: 


Impacts on, for example, public awareness, 
attitudes, understanding or behaviour that 
arose from engaging the public with research. 
In these cases, the submitting unit must show 
that the engagement activity was, at least in 
part, based on the submitted unit's research 
and drew materially and distinctly upon it. 
Further guidance and examples are set out in 
the ‘Panel criteria’, Annex A. 


July 2018 


274. The panels also acknowledge that there are multiple and diverse pathways through which research 
achieves impact. Impact may be the result of individual or collective research (or a combination of these) within 
or between a range of organisations, within higher education and beyond, including collaboration beyond the 
UK. The associated impact may be achieved by a variety of possible models: from individuals, to inter- 
institutional groups, to groups including both academic and non-academic participants. The relationship 
between research and impact can be indirect or non-linear. The impact of research may be foreseen or 
unforeseen. It can emerge as an end product, but can also be demonstrated during the research process. Impact 
takes place through a wide variety of mechanisms. It may effect change or enrichment for local, national or 
international communities, groups or individuals. Consequently, public engagement may be an important feature 
of many case studies, as the mechanism by which the impact claimed has been achieved. 


288. Engaging the public with the submitting unit’s research (for example, through citizen science, patient and 
public involvement in health, or through public and community engagement), is an activity that may lead to 
impact. Sub-panels will welcome, and assess equitably, case studies describing impacts achieved through public 
engagement, either as the main impact described or as one facet of a wider range of impacts. Panels expect that 
case studies based on public engagement will demonstrate both reach (e.g. through audience or participant 
figures) and significance, and will take both into account when assessing the impacts. Examples of impacts 
arising from public engagement can be found as part of Table 1 (Annex A). 


Annex A: Examples of impact achieved through public engagement are integrated into the different areas of 
impact in Table 1. More detailed advice on achieving and evidencing impact through public engagement can be 
found on the website of the National Coordinating Centre for Public Engagement: 
http://www.publicengagement.ac.uk/. 

4. Examples are also provided of impact evaluation frameworks used outside higher education. Impact partners 
may also have their own evaluation frameworks that could be drawn upon to evidence impact. 


www.publicengagement.ac.uk 5 


K . In the remainder of this document we highlight the key issues that we have identified with the consultation 
ey Issues documents, and outline the headlines of how we intend to respond. |We have highlighted in yellow questions \ 
which we hope you might consider in advance of the event, or discuss with colleagues 


Equality and diversity 


Currently the Guidance and Criteria frame Equality and Diversity almost exclusively in terms of internal HE staff policies and the ‘internal’ conduct of research. There are very significant 
E&D issues linked to the generation of impact. There is only one point at which this is acknowledged: 


‘Reach will be understood as the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact’ Draft Criteria, paragraph 276) 


We will argue that the E&D issues related to impact need to be surfaced and addressed in the guidance, for instance in the Environment section. We will argue that this is a very 
significant omission, and provide suggestions for how this could be approached. 


Assessment of impact 


We will argue that only assessing the ‘reach and significance’ of impact, and not the ‘rigour’ of the processes described or of the evaluation undertaken, significantly weakens the 
robustness of the process. 


This section should provide a narrative, with supporting evidence, to explain: 


e How the research underpinned (made a distinct and material contribution to) the impact (the rigour of the methods used to maximise the impact of the underpinning research and/or 
to secure evidence of how that impact was manifested) 


e The nature and extent of the impact (the reach and significance of the impact) 


Rigour will be understood as the extent to which the methods linking the research to the claimed impacts demonstrate intellectual coherence and robustness, sensitivity and adaptation 
to their social context (including equality and diversity) and enable the adoption of the most robust and appropriate evaluative techniques 


We note that there is virtually no mention of the evaluation of impact in either document, and we will propose a short additional section that provides guidance about the value of 
evaluation in securing robust evidence of impact. 


Consistency 

We will argue that while significant steps have been taken to increase the consistency of the guidance there is more that could be done to iron out inconsistency and to increase its 
robustness. In particular, we will identify significant inconsistency in the ways in which indicators of impact are articulated and deployed in the guidance and draft criteria. There are 
various different formulations of what count as indicators of impact. For instance: 


The draft guidance states: 
‘Impact includes, but is not limited to, an effect on, change or benefit to: 
e the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding’ 


The draft criteria list: 
‘performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries’ (277). 


The draft guidance, in its commentary about public engagement, states: ‘impacts on for example public awareness, attitudes, understanding or behaviour...’ (p.90, para 139) 


The indictors of impact listed in Annex A of the draft criteria draw on these and other possible indicators in an unsystematic way. 


We will argue that it would be helpful if there was a more consistent and coherent articulation of possible indicators, that draws on established approaches to evaluation particularly 
in the health sector. We will suggest that it would be useful to differentiate between the following types of indicator (and to use these distinctions consistently): 

e Structural indicators: e.g. policies, systems and infrastructure 

e Process indicators: e.g. performance, activity 

e Outcome indicators: e.g. skills, attitudes, understanding, behaviour, well-being 


It is notable that the indicators listed in Annex A (Draft Criteria) for the different Areas of Impact are dominated by structural and process indicators, and many of the Areas fail to 
include any outcome indicators at all. The Area of ‘Understanding, Learning and Participation’, however, is dominated by outcome indictors, at the expense of the other types, 
making this area appear qualitatively different to the others. In effect, it appears to have been used as a convenient place to locate impacts arising from public engagement, which 
will have unintended consequences: firstly, it will suggest that public engagement is primarily about achieving individual outcomes for people, whereas in practice it can make a 
significant contribution to both structures and processes in society (for instance through public dialogue or co-production); and secondly, it will discourage people from considering 
the value of public engagement across all nine Areas of Impact. 


To address this we will argue that: 

(a) ‘Understanding, Learning and Participation’ needs to be re-focused so it is consistent with the other 8 Areas —i.e. capturing a broad coalition of organisations and activity focused 
on a meaningful cluster of social outcomes. We will suggest this might be framed as ‘Education, Skills and Community Development’ 

(b) All nine areas should include examples of Structural, Process and Outcome indicators, expressed in a more consistent way. Slides 8 & 9 offer some exemplar indicators 


What does all this mean for public engagement? 


While we are pleased to see PE acknowledged as an important pathway to impact, we think there are elements of the current framing of the guidance and criteria which will mitigate 
against the full potential of PE being realised, and robustly assessed. 


We will identify the following broad recommendations: 


e Itis vital that stronger signals are sent to the sector about the need take account of E&D in reporting on their impact activities, not just in their staff policies. An obvious hook for 
this is in the definition of Impact, which includes ‘the reduction or prevention of harm, risk, cost of other negative effects’. There should be a clear expectation set by the panels that 
equality and diversity will be addressed in impact planning (in the ‘environment’), and guidance about what will count as evidence of impact (in terms of both ‘reach’ and 
‘significance’) within impact case studies. 

e A greater emphasis on rigour and evaluation is vital if we want to see a step change in the quality of submissions and the robustness of assessment in 2021. This needs to be 
signalled unequivocally. Including ‘rigour’ in the framing of impact assessment would send such a signal, as would clearer guidance about the use of evaluation to provide robust 
evidence 

e The lack of consistency with which the guidance and criteria currently frame indicators of impact risks placing public engagement in a ‘silo’. By describing these indicators in a more 
coherent and consistent way, it will be possible to demonstrate how public engagement can contribute value across all nine Areas of Impact. 


To back these up we will provide 
e Suggested wording and links to relevant frameworks and tools to focus attention on how E&D issues should be foregrounded and factored into the guidance and criteria 
e Concrete suggestions for how meaningful indicators of impacts arising from public engagement can be ‘wired in’ across the nine Areas of Impact 


The next 2 slides outline our current thinking about how to do the latter: 

e They try to provide a consistent set of descriptions of the nine Areas of Interest 

e They suggest how members of the public can be meaningfully engaged in each of these areas 

e They describe some generic indicators which we think would help provide greater consistency and coherence, and identify some indicators specifically focus upon impacts arising 
from public engagement. 
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Integrating public engagement across all areas 


Area of impact / domain 
Impacts on creativity, 


culture and society 


Impacts on the 
environment 


Impact on social welfare 


Impacts on production 


Impacts on commerce 
and the economy 


Impacts on practitioners 
& professional services 


Impacts on public policy, 
law and services 


Impacts on 
health, wellbeing 
& animal welfare 


Impacts on education, 
lifelong learning and 
community engagement 


Motivation 


To stimulate participation in the 
arts and culture; to support creative 
expression 


To enhance the environment, 
address climate change, 
stewardship, conservation 


To address poverty and inequality 
and other social challenges 


To create new, or improve existing 
products and production processes 


To generate wealth, create 
efficiencies, improve financial 
regulation 


To enhance professional practice & 
deliver high-quality services that 
meet ‘users’ needs 


To create effective, just and fair 
polices, regulations and laws 


To improve human and animal 
health and wellbeing 


To stimulate lifelong learning, 
citizenship and participation across 
society 


Professional communities 


People working in arts and cultural 
sector 


People working in all areas of society, 
committed to positive impacts on the 
environment 


People working in social care, and to 
address poverty social exclusion and 
social mobility 


People working in manufacturing and 
on the production of new products 


People working in financial services 
and regulation, and in businesses with 
a ‘bottom line’ 


People working to deliver high quality 
services, in any sector 


Government (national, devolved and 
local); regulators and law makers 


People working in & allied to 
healthcare, social care or animal 


welfare to deliver services; companies 


supplying products & services, carers 
& community initiatives 


People working in formal and informal 


education and community 
development 


Public roles 


Audiences; participants; 
communities of place and interest; 
volunteers; supporters 


Citizen scientists; activists; 
volunteers; communities of place; 


Citizens; community members; 
carers 


Consumers; employees; union 
members 


Consumers; customers; employees; 
communities of place; citizens 


Service users; customers; members; 
trustees and governors; clients 


Citizens; consumers; community 
members; voters 


Patients; carers; parents; pet owners 


Learners; community members; 
pupils; parents 
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This table attempts to provide a more consistent and coherent framing of the 9 areas of impact, and to model how 
PE contributes to each 


Contribution of public engagement? 


Stimulate involvement in arts and culture; widen 
participation and representation; support development of 
services better tuned to users 


Stimulate behaviour change; build understanding and 
awareness; contribute to public debate 


Contribute to public debate; support development of 
services better tuned to users; inform policy 
development; address behaviours 


Hold to account for business practices; contribute to 
shaping of new products so they better meet customer 
needs 


Hold to account for business practices; contribute to 
shaping of new products so they better meet customer 
needs; stimulate innovation 


Hold to account for quality of service; contribute to 
shaping of new services so they better meet user needs; 
embed reflective practices 


Contribute to public debate; support development of 
policies, governance and laws; inform policy development 
through dialogue 


Stimulate awareness and understanding; address 
behaviours; enhance services through patient 


involvement; improved prevention, treatment and 
support 


Stimulate take up of lifelong learning; encourage 
participation in civic life; enhance services through user 


engagement 


. . This slide offers a set of exemplar indicators which could be adapted for each domain, with examples of indicators 
Integrating public engagement across all areas linked to impacts arising from public engagement 


Impact domain Structural indicators Process indicators Outcome indicators 


Evidence of influence on policies, Evidence adoption of new or changed methods, models, techniques and Evidence of increased awareness and 
guidelines, standards or regulations approaches (e.g. enhanced co-production; changes to resource-use) understanding 
pertaining to each domain Evidence of influence on project activity and operations, product development Evidence of increased skills 
Evidence of influence on decision and/or service delivery Evidence of increased empathy or tolerance 
making and accountability Evidence of challenge to ‘received wisdom’ or established paradigms and Evidence of increased creativity 
arrangements (e.g. governance assumptions Evidence of increased educational attainment 
arrangements) Evidence of influence on capacity building activity (e.g. CPD; curricula; teaching Evidence of increased confidence 
Evidence of influence on the strategic methods) Evidence of increased motivation 
objectives of an organisation Evidence of changes to working practices, e.g. through increased reflective Evidence of increased trust 
Evidence of influence on arrangements practice Evidence of enhanced health and wellbeing 
to support collaboration and Evidence of influence on decision-making outcomes (e.g. planning decisions) Evidence of enhanced quality of life 
partnership working (e.g. formal Evidence of changes to extent, quality and longevity of collaboration Evidence of influence on social mobility 
partnership arrangements) Evidence of changes to management practices Evidence of improved equality and diversity 
Evidence of adoption of regulations and Evidence of decision making processes being influenced outcomes 
routines to enhance social Evidence of changes to progression opportunities for individuals with protected Evidence of strengthened or extended 
responsibility, ethical practice, equality characteristics networks 
and diversity and sustainability Evidence of improved access to resources or opportunities Evidence of changed behaviours 
Evidence of positive environmental Evidence of uptake of research in public debate and discourse Evidence of increased ‘user’ satisfaction with 
benefits Evidence of the identification and mitigation of risks, and hazards products or services 
Evidence of influence on consideration of ethics and values Evidence of enhanced patient outcomes 
Evidence of reduced costs, greater profitability, enhanced efficiency or 
productivity 
Evidence of improved accountability / monitoring 


Creativity & 
culture 
Environment 
Social welfare 
Production 
Commerce and 
the economy 
Practitioners & 
prof services 
Public policy, 
law & services 
Health, 
wellbeing & 
animal welfare 
Education, 
lifelong learning 
and community 
engagement 


Indicators potentially linked to engagement of the public 
Structural indicators Process indicators Outcome indicators 


Evidence of research process leading to enhanced public involvement in the e Evidence of how specific individuals or groups 
shaping and delivery of services have been directly affected in all indicator areas 


Evidence of discussion and debate being stimulated in the public sphere 
Evidence of individual / group decision making being influence by research 


e Evidence of research process involving 
the public in the shaping of new 
policies, guidelines and regulations 
Evidence of research process resulting 
in public involvement becoming 


Evidence of the increased uptake of lifelong learning Evidence of increased 
public participation in the different domains 

Evidence of widened participation and access (e.g. by marginalized, under- 
engaged and/or diverse audiences 


embedded in governance and 
accountability arrangements 

Evidence of research process ensuring 
public ‘voice’ holds organisations to 
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What else do you think needs to be addressed in our response? 


Please come along prepared to share other significant issues or concerns with the draft guidance which you think we should address in our response 


Please also come prepared to talk about the aspects of the draft guidance which you find particularly helpful and constructive that you think we should comment 
positively upon 


We will devote part of the event to considering the wider issue of sector capacity building to ensure, collectively, we ‘rise to the challenge’ of the REF, and 
maximise the opportunities it presents to embed higher quality engagement with the public in the research community. Please come prepared to share your 
views of: 

e The particular barriers / blocks to the integration of high quality public engagement in REF submissions 

e The ‘triggers’ or methods you have deployed which have been successful 

e Ideas for how the NCCPE and other agencies might provide well targeted support 


With thanks to everyone who contributed to drafts of this paper 


Please direct any enquiries to nncpe.enguiries@ uwe.ac.uk 
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Centre for 


National 
| | | Co-ordinating 
| Public Engagement 


The National Co-ordinating Centre for Public Engagement (NCCPE) is 
internationally recognised for its work supporting and inspiring universities 
to engage with the public. 


We work to change perspectives, promote innovation, and nurture and 
celebrate excellence. We also champion meaningful engagement that makes 
a real and valued difference to people’s lives. 


The NCCPE is supported by the UK Higher Education Councils, Research 
Councils UK and Wellcome, and has been hosted by the University of Bristol 
and the University of the West of England since it was established in 2008. 


Ele University of UWE 
BRISTOL 


A 
Bristol | bean 


WWW public neag 
ement.ac.uk 


National Co-ordinating Centre 
for Public Engagement 


2nd Floor, Arnolfini 
16 Narrow Quay 
Bristol, BS1 4QA 


Tel 0117 328 7190 
Email nccpe.enquiries@uwe.ac.uk 
Twitter @NCCPE 
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